跳转至

激进解码期 KV 淘汰的关键要素：时间聚合与排序保持

文章背景与核心概要

随着大语言模型上下文窗口的不断扩展，KV 缓存（Key-Value Cache）的显存占用成为了长文本推理的严重瓶颈。当前的解码期 KV 缓存压缩研究往往将主要精力放在令牌评分函数（token scoring functions）的设计上，而将时间聚合规则（跨解码步骤如何组合分数）视为微不足道的实现细节。

本文深入探讨了激进 KV 压缩底下的核心影响因素。研究表明，指数移动平均（EMA）等时间聚合规则会耦合层权重与时间保留，使得大致保持顺序的评分器修改在淘汰集层面上几乎无法区分。同时，保持评分稳定性（如 Value-norm 及其变体）对于防止性能急剧下降至关重要。基于这些发现，作者提出了 InertiaKV & InertiaKV-Lazy 动态淘汰方法，并通过周期性刷新大幅提升了解码吞吐量；此外还创新性地提出了无分数字解码（Score-Free Decoding）方案，在几乎不损失性能的前提下彻底消除了生成过程中的评分延迟。

Current research on decoding-time Key-Value (KV) cache compression heavily prioritizes the design of token scoring functions, while the temporal aggregation rules (how scores are combined across decoding steps) are usually treated as minor implementation details.

This paper demonstrates that under aggressive KV compression: 1. Temporal Aggregation Matters: Exponential-moving-average (EMA) aggregation couples layer weighting and temporal retention, making approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level. 2. Scorer Stability: Value-norm and entropy variants maintain high correlation with attention and yield nearly unchanged retention sets. Conversely, metrics like KeyDiff, key norm, recency, and learned scorers alter the ranking and degrade performance significantly. 3. Proposed Solutions: - InertiaKV & InertiaKV-Lazy: New EMA-based decoding-time eviction methods. The lazy, periodic-refresh variant (InertiaKV-Lazy) achieves a 1.34–1.46× decoding throughput speedup compared to full-refresh InertiaKV. - Score-Free Decoding: A novel empirical operating point where the full context is scored once at the first decode step and the ranking is frozen thereafter. This removes all subsequent scoring overhead while incurring an average quality change of only +0.03 across benchmarks.

Evaluated across six open-weight backbones on LongBench, LongBench-v2, and RULER benchmarks.

核心亮点与贡献

Key Highlights & Contributions

  • 设计因素解耦： 明确了时间聚合和排序保持是 KV 淘汰中至关重要且相互独立的 설계 因素（尽管评分质量在总体上仍然重要）。
  • 吞吐量显著提升： 引入了 InertiaKV-Lazy，在不牺牲保留质量的前提下显著提升了解码效率。
  • 无分数基线： 证明了在初始步骤后冻结上下文评分只会引入微不足道的性能下降，同时完全消除了生成过程中的评分延迟。
  • Distinct Design Factors: Identifies temporal aggregation and ranking preservation as critical, distinct design factors in KV eviction (though scoring quality remains important generally).
  • Throughput Improvements: Introduces InertiaKV-Lazy, significantly boosting decoding efficiency without sacrificing retention quality.
  • Score-Free Baseline: Demonstrates that freezing context scoring after the initial step introduces negligible performance drops while completely eliminating scoring latency during generation.

全文与资源

Full-Text & Resources

license icon view license